Depth information based optical distortion correction circuit and method

ABSTRACT

Provided are a depth information based optical distortion correction (ODC) circuit and method. The ODC circuit includes a depth acquisition unit acquiring a depth of an image, a grid generation unit dynamically generating a correction grid corresponding to the depth of the image using depth information of the image and a projection matrix, and a distortion correction unit correcting optical distortion in the image using the correction grid. The corrected image can then be stored, transmitted, or otherwise output. The image can be a single static image or photograph or a frame in a video recording.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority from Korean Patent Application No.10-2013-0110476 filed on Sep. 13, 2013 in the Korean IntellectualProperty Office, and all the benefits accruing therefrom under 35 U.S.C.119, the contents of which are incorporated herein by reference in itsentirety.

FIELD

The present invention relates to a depth information based opticaldistortion correction circuit and method.

BACKGROUND

Optical distortion (ODC), also referred to as lens distortioncorrection, is used to correct optical distortion in an image generatedthat is attributable to curvature characteristics of a lens. A generaloptical distortion correction circuit stores a correction grid andcorrects optical distortion in the image using the stored correctiongrid. Alternatively, the general optical distortion correction circuitmay store a plurality of correction grids in the form of a look-up table(LUT) and may correct optical distortion in each of a plurality ofimages, including images having various depths, using the storedcorrection grids.

However, when the same correction grid is used for different images thathave different extents of optical distortion due to their differentdepths (e.g., far and near), accuracy of correcting optical distortionmay be lowered. In addition, even if a plurality of correction grids arestored in the LUT, it is not possible to store the correction gridscorresponding to all possible depths; consumption of a memory space forstoring the correction grids may increase the greater the number ofcorrection grids to be stored.

SUMMARY

The present invention, comprising a depth module and a grid correctionmodule, supports the dynamic grid correction by using depth parameterswithout the need for various LUTs (look up tables). As a result, thepresent invention can reduce the necessary grid memory and obtain highaccurate grid for any kind of scene.

The present invention provides a depth information based opticaldistortion correction (ODC) circuit and method, which can dynamicallygenerate correction grids corresponding to various depths of an image.

The present invention also provides a depth information based opticaldistortion correction circuit and method, which can increase accuracy ofcorrecting optical distortion by generating correction gridscorresponding to all possible depths.

The present invention also provides a depth information based opticaldistortion correction circuit and method, which can reduce consumptionof a memory space for storing correction grids.

These and other objects of the present invention will be described in orbe apparent from the following description of the preferred embodiments.

According to an aspect of the present invention, there is provided adepth information based optical distortion correction (ODC) circuit. TheODC circuit includes a depth acquisition unit configured to acquire adepth of an image, a grid generation unit configured to dynamicallygenerate a correction grid corresponding to the depth of the image usingdepth information of the image and a projection matrix, and a distortioncorrection unit configured to correct optical distortion in the imageusing the correction grid.

In various embodiments, the grid generation unit may be configured togenerate coordinates of the correction grid using coordinates of anoriginal grid, the depth information of the image, and the projectionmatrix.

In various embodiments, the ODC may further comprise a projection matrixcalculation unit configured to calculate the projection matrix using: anoptical distortion in at least two images captured by an image sensorand having different depths; and depth information of the at least twoimages.

In various embodiments, the ODC circuit may further comprise adistortion calculation unit configured to compare the at least twoimages with the original grid and to calculate the optical distortion ofeach of the at least two images.

In various embodiments, the depth acquisition unit is configured toacquire the depths of the at least two images.

In various embodiments, the ODC circuit may further comprise a storingunit storing the projection matrix.

In various embodiments, the depth acquisition unit may be configured toacquire a depth of at least one frame among a plurality of framescaptured by an image sensor.

In various embodiments, the depth acquisition unit may be configured toacquire the depth of the image using a stereo depth extractionalgorithm.

In various embodiments, the depth acquisition unit may be configured toacquire the depth of the image using an auto focus (AF) algorithm.

In various embodiments, the depth acquisition unit may be configured toacquire the depth of the image provided from a time of flight (TOF)sensor.

In various embodiments, the depth acquisition unit may be configured toacquire the depth of the image input by a user.

According to another aspect of the present invention, there is provideda depth information based optical distortion correction (ODC) method.The ODC method includes acquiring a depth of an image; dynamicallygenerating a correction grid corresponding to the depth of the imageusing the depth information of the image and the projection matrix; andcorrecting optical distortion in the image using the correction grid.

In various embodiments, the dynamically generating of the correctiongrid corresponding to the depth of the image may comprise dynamicallygenerating coordinates of the correction grid using coordinates of anoriginal grid, the depth information of the image, and the projectionmatrix.

In various embodiments, the ODC method may further comprisepre-processing to generate the projection matrix, wherein thepre-processing can comprise: acquiring at least two images havingdifferent depths; and calculating the projection matrix using opticaldistortion in the at least two images and depths of the at least twoimages.

In various embodiments, the pre-processing may further comprise storingthe projection matrix in a storage media.

In accordance with another aspect of the invention, provide is animaging system with depth information based optical distortioncorrection (ODC). The system comprises a memory having stored therein aprojection matrix; an image sensor configured to receive an opticalimage signal from an image captured through at least one lens and toconvert the received optical image signal into an electrical imagesignal; an image signal processor configured to process the electricalimage signal into a distortion corrected optical image. The image signalprocessor comprises a depth acquisition unit configured to acquire adepth of the image; a grid generation unit configured to dynamicallygenerate a correction grid corresponding to the depth of the image usingdepth information of the image and the projection matrix; and adistortion correction unit configured to correct optical distortion inthe image using the correction grid. The system further comprises adisplay controller configured to output the corrected optical image to adisplay.

In various embodiments, the grid generation unit may be configured togenerate coordinates of the correction grid using coordinates of anoriginal grid, the depth information of the image, and the projectionmatrix.

In various embodiments, the system may comprise a projection matrixcalculation unit configured to calculate the projection matrix using anoptical distortion in at least two images captured by an image sensorand having different depths and depth information of the at least twoimages.

In various embodiments, the system may further comprise a distortioncalculation unit configured to compare the at least two images with theoriginal grid and to calculate the optical distortion of each of the atleast two images.

In various embodiments, the depth acquisition unit is configured toacquire the depth of the image using an auto focus (AF) algorithm; atime of flight (TOF) sensor; or by an input by a user.

In various circuits, methods, and/or systems in accordance with aspectsof the present invention, the image may be a static image or photographor a frame of a video image.

In various circuits, methods, and/or systems in accordance with aspectsof the present invention, the image processing and image distortioncorrection may be in real time or near real time.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other features and advantages of the present inventionwill become more apparent by describing in detail preferred embodimentsthereof with reference to the attached drawings in which:

FIG. 1 is a block diagram of an embodiment of a depth information basedoptical distortion correction (ODC) circuit according to aspects of thepresent invention;

FIG. 2 is a block diagram of another embodiment of a depth informationbased ODC circuit according to aspects of the present invention;

FIG. 3 is a block diagram of still another embodiment of a depthinformation based ODC circuit according to aspects of the presentinvention;

FIG. 4 is a block diagram illustrating an embodiment of a projectionmatrix calculating operation performed by some embodiments of a depthinformation based ODC circuits according to aspects of the presentinvention;

FIG. 5 is a conceptual diagram illustrating an embodiment of anoperation or method of acquiring a first grid image and a second gridimage in FIG. 1, according to aspects of the present invention;

FIG. 6 is a block diagram of an embodiment of a system on chip includingan embodiment of depth information based ODC circuit according toaspects of the present invention;

FIG. 7 is a block diagram of an embodiment of a user system including adepth information based ODC circuit according to aspects of the presentinvention;

FIG. 8 is a block diagram of another embodiment of a user systemincluding a depth information based ODC circuit according to aspects ofthe present invention;

FIG. 9 is a block diagram of still another embodiment of a user systemincluding a depth information based ODC circuit according to aspects ofthe present invention;

FIG. 10 is a flowchart illustrating an embodiment of a depth informationbased ODC method according to aspects of the present invention;

FIG. 11 is a flowchart specifically illustrating an embodiment of thepre-processing function shown in FIG. 10; and

FIG. 12 is a flowchart specifically illustrating an embodiment of theoptical distortion correction function shown in FIG. 10.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Aspects of the present invention will now be described more fullyhereinafter with reference to the accompanying drawings, in whichpreferred embodiments of the invention are shown. This invention may,however, be embodied in different forms and should not be construed aslimited to the embodiments set forth herein. The same reference numbersindicate the same components throughout the specification, unlessotherwise indicated or understood. In the attached figures, thethickness of layers and regions is exaggerated for clarity.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing the invention (especially in the context of thefollowing claims) are to be construed to cover both the singular and theplural, unless otherwise indicated herein or clearly contradicted bycontext. The terms “comprising,” “having,” “including,” and “containing”are to be construed as open-ended terms (i.e., meaning “including, butnot limited to,”) unless otherwise noted.

Unless defined otherwise, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs. It is noted that the use of anyand all examples, or exemplary terms provided herein is intended merelyto better illuminate the invention and is not a limitation on the scopeof the invention unless otherwise specified. Further, unless definedotherwise, all terms defined in generally used dictionaries may not beoverly interpreted.

Aspects of the present invention will be described with reference toperspective views, cross-sectional views, and/or plan views, in whichpreferred embodiments of the invention are shown. Thus, the profile ofan exemplary view may be modified according to manufacturing techniquesand/or allowances. That is, the embodiments in accordance with theinvention are not intended to limit the scope of the present invention,but cover all changes and modifications that can be caused due to achange in manufacturing process. Thus, regions shown in the drawings areillustrated in schematic form and the shapes of the regions arepresented simply by way of illustration and not as a limitation.

Hereinafter, an embodiment of a depth information based opticaldistortion correction (ODC) circuit according to aspects of the presentinvention will be described with reference to FIGS. 1 to 3.

FIG. 1 is a block diagram of an embodiment of a depth information basedoptical distortion correction (ODC) circuit according to aspects of thepresent invention.

Referring to FIG. 1, the ODC circuit 100 includes a depth acquisitionunit 110, a grid generation unit 120, and a distortion correction unit130.

The depth acquisition unit 110 may be configured to acquire a depth froma distorted image, whether a static image or a frame of a video. Forexample, the depth acquisition unit 110 may acquire the depth from thedistorted image using a stereo depth extraction algorithm or an autofocus (AF) algorithm. The depth acquisition unit 110 may be suppliedwith the depth of the distorted image from a time of flight (TOF)sensor. However, the present invention does not limit the method ofacquiring the depth of distorted image to those stated herein. The depthacquisition unit 110 may acquire the depth from the distorted imageusing a variety of methods, such as those well known in the art.

In a case of photographing a still image, the depth acquisition unit 110may acquire a depth of an image photographed by an image sensor (notshown). In a case of recording a video, the depth acquisition unit 110may acquire a depth of at least one frame among a plurality of framescaptured by an image sensor. While the video is being recorded, thedepth acquisition unit 100 may acquire depths of frames captured by theimage sensor on a real-time basis.

The grid generation unit 120 may be configured to dynamically generate acorrection grid corresponding to the depth of a distorted image. Thegrid generation unit 120 may receive depth information of a distortedimage from the depth acquisition unit 110. The grid generation unit 120may dynamically generate a correction grid using the depth informationof the distorted image and the projection matrix 121.

As expressed in Equation (1) below, the grid generation unit 120 maydynamically generate coordinates of the correction grid by performingoperations of or applying the projection matrix 121, the coordinates ofthe original grid, and the depth of the distorted image.

$\begin{matrix}{\begin{bmatrix}u \\v \\1\end{bmatrix} = {P\begin{bmatrix}x \\y \\z\end{bmatrix}}} & (1)\end{matrix}$where u and v indicate coordinates of the correction grid, x and yindicate coordinates of the original grid, z indicates the depth of thedistorted image, and P indicates the projection matrix. The coordinatesof a grid may include coordinates of grid points (e.g., x,y,z).

The projection matrix 121 may be pre-stored in the grid generation unit120. The projection matrix 121 may be defined by and represent curvaturecharacteristics of a lens.

The distortion correction unit 130 may receive the distorted image andcorrect optical distortion in the distorted image to then output acorrected image with little or no distortion from the curvature of thelens. The distortion correction unit 130 may receive the correction gridfrom the grid generation unit 120, wherein the distortion correctionunit 130 may automatically correct the optical distortion in thedistorted image using the correction grid.

Since optical distortion correction using a correction grid is generallyknown to one skilled in the art, a detailed description thereof will beomitted.

FIG. 2 is a block diagram of another embodiment of a depth informationbased optical distortion correction (ODC) circuit according to aspectsof the present invention. For the sake of convenient explanation, thefollowing description will focus on differences from the ODC circuit 100shown in FIG. 1.

Referring to the embodiment of FIG. 2, the ODC circuit 200 furtherincludes a storing unit 140, when compared to the ODC circuit 100 shownin FIG. 1.

In the ODC circuit 200 embodiment of FIG. 2, a projection matrix 121 maybe pre-stored in the storing unit 140. The storing unit 140 may includenon-transitory storage circuit, for example, non-volatile memory, suchas an electrically erasable programmable read only memory (EEPROM) or aflash memory, but aspects of the present invention are not limitedthereto.

The grid generation unit 120 may access the storing unit 140 to refer tothe projection matrix 121. The grid generation unit 120 may dynamicallygenerate a correction grid using the depth information received from thedepth acquisition unit 110 and the projection matrix 121 referred to bythe storing unit 140.

FIG. 3 is a block diagram of another embodiment of a depth informationbased optical distortion correction (ODC) circuit according to aspectsof the present invention. For the sake of convenient explanation, thefollowing description will focus on differences from the ODC circuit 200shown in FIG. 2.

Referring to the embodiment of FIG. 3, in the ODC circuit 300, a depthof a distorted image is input by a user, unlike in the ODC circuit 200shown in FIG. 2.

The user may manually input the depth of the distorted image using avariety of well-known input circuits, such as a keyboard, a mouse, abutton, a keypad, or a touch screen. A depth acquiring unit 110 mayacquire the depth input by the user.

As described above, in the ODC circuits according to some embodiments ofthe present invention, depths of first and second images are acquired,first and second correction grids corresponding to the depths of thefirst and second images are generated, and optical distortion in each ofthe first and second images may be corrected using the first and secondcorrection grids. The depths of the first and second images may rangefrom 0 (zero) to infinity (∞), or an approximation thereof. That is tosay, in the ODC circuits according to some embodiments of the presentinvention, since correction grids corresponding to various depths ofimages are dynamically generated, accuracy of correcting opticaldistortion can be increased and consumption of a memory space forstoring the correction grids can be reduced.

Meanwhile, in the ODC circuits according to some embodiments of thepresent invention, pre-processing may be required to calculate theprojection matrix and to store the projection matrix. Hereinafter, thepre-processing of the ODC circuits according to some aspects of thepresent invention will be described with reference to the embodiments ofFIGS. 4 and 5.

FIG. 4 is a block diagram illustrating an embodiment of a projectionmatrix calculating operation performed by depth information basedoptical distortion correction (ODC) circuits according to some aspectsof the present invention and FIG. 5 is a conceptual diagram illustratingan embodiment of an operation of method of acquiring a first grid imageand a second grid image in FIG. 1.

Referring to the embodiment of FIG. 4, a distortion calculation unit 150may receive first and second grid images.

As illustrated in the embodiment of FIG. 5, an image sensor (not shown)may photograph original grids 20 and 30, respectively positioned atfirst and second distances to acquire first and second grid imageshaving different depths. In FIG. 5, reference numeral 10 denotes a lens.Due to curvature characteristics of the lens, optical distortion may begenerated in each of the first and second grid images. Here, the firstdistance may be shorter than the second distance with respect to thelens. Alternatively, the first depth of the first grid image may besmaller than the second depth of the second grid image. Therefore, theextent of optical distortion in the second grid image may be greaterthan the extent of optical distortion in the first grid image.

Referring again to the embodiment of FIG. 4, the distortion calculationunit 150 may be configured to compare the first and second grid imageswith an original grid image and may calculate optical distortion in eachof the first and second grid images. The distortion calculation unit 150may calculate a deviation in distances between centers of the images andgrid points as the extent of optical distortion. The original grid imagemay be stored in the distortion calculation unit 150 or in an externaldevice (e.g., the storing unit 140) to then be supplied to thedistortion calculation unit 150. Throughout the specification, theoptical distortion in the first grid image and the optical distortion inthe second grid image are defined as first distortion and seconddistortion, respectively, which will be described below in detail.

The depth acquisition unit 160 may acquire depths of the first andsecond grid images. As described above, the depth acquisition unit 160may acquire the depths of the first and second grid images using astereo depth extraction algorithm or an auto focus (AF) algorithm. Thedepth acquisition unit 160 may be supplied with the depths of the firstand second grid images from a time of flight (TOF) sensor.Alternatively, the depth acquisition unit 160 may acquire the depths ofthe first and second grid images input by a user. Throughout thespecification, the depths of the first and second grid images aredefined as first and second depths, respectively, which will bedescribed below in detail.

The projection matrix calculation unit 170 is configured to calculatethe projection matrix using the first distortion and the seconddistortion and information about the first and second depths. Thecalculated projection matrix may be transmitted to the grid generationunit 120 to then be stored therein or transmitted to the storing unit140 to then be stored therein.

In FIG. 4, barrel distortion is exemplified as the form of opticaldistortion being calculated and corrected for, but aspects of thepresent invention are not limited thereto. For example, pin-cushiondistortion may also be calculated by the distortion calculation unit 150and corrected for.

According to embodiments, the distortion calculation unit 150, the depthacquisition unit 160, and the projection matrix calculation unit 170,having been described with reference to FIG. 4, may be provided asinternal components of the various embodiments of ODC circuits describedherein, and otherwise able to be manufactured according to aspects ofthe present invention. The depth acquisition unit 110 (e.g., in FIGS.1-3) and the depth acquisition unit 160 (e.g., in FIG. 4) may beprovided as a single component. In this case, the ODC circuits accordingto some embodiments of the present invention may perform thepre-processing at the time of initial configuration or lens exchange.

Alternatively, the distortion calculation unit 150, the depthacquisition unit 160 and the projection matrix calculation unit 170,having been described with reference to the embodiment of FIG. 4, may beprovided as components of an external device, and the ODC circuitsaccording to some embodiments of the present invention may receive thecalculated projection matrix from the external device. In addition, instages of fabricating the ODC circuits according to some embodiments ofthe present invention, the projection matrix may be stored in the gridgeneration unit 120 or the storing unit 140.

FIG. 6 is a block diagram of an embodiment of a system on chip (SOC)including depth information based optical distortion correction (ODC)circuits, according to aspects of the present invention.

Referring to the embodiment of FIG. 6, the system on chip 1000 mayinclude a core processor (CORE) 1100, a memory 1200, a displaycontroller 1300, an image sensor 1400, an image signal processor (ISP)1500, an interface 1600, and a data bus 1700.

The core 1100, the memory 1200, the display controller 1300, the imagesensor 1400, the ISP 1500, and the interface 1600 may be directly orindirectly connected to each other through the data bus 1700. The databus 1700 may correspond to a path through which data moves, e.g.,electrically conductive traces, wires or connections or optical cables,wires, fibers, or connectors.

The core 1100 may include one processor core (single-core) or aplurality of processor cores (multi-core) to process data. For example,the core 1100 may be a multi-core, such as a dual-core, a quad-core or ahexa-core. The core 1100 may further include a cache memory positionedinside or outside.

The memory 1200 may be configured to store data processed by the core1100 and/or the ISP 1500 and/or programs executed by the core 1100and/or the ISP 1500. The memory 1200 also may be configured to storedata of, from, or representing images photographed by the image sensor1400. The memory 1200 may include one or more volatile memories, such asa double data rate static DRAM (DDR SDRAM) or a single data rate SDRAM(SDR SDRAM), and/or one or more non-volatile memories, such as anelectrical erasable programmable ROM (EEPROM) or a flash memory, asexamples.

The display controller 1300 may control a display device to allow thedisplay device to display an image.

The image sensor 1400 may photograph (or capture) an image, wherein theimage sensor 1400 may receive an optical image signal through a lens(not shown) and may convert the received optical image signal into anelectrical image signal. As examples, the image sensor 1400 may includea charge coupled diode (CCD) image sensor or a complementary metal oxidesemiconductor (CMOS) image sensor.

The ISP 1500 may process the image signal photographed by the imagesensor 1400. The ISP 1500, including each of the ODC circuits 100 to 300according to some embodiments of the present invention, may correctoptical distortion in the image photographed by the image sensor 1400.In some embodiment, the ISP 1500 may include the ODC circuit 100 shownin FIG. 1, as exemplified in the embodiment of FIG. 6. However, in otherembodiments, the ISP 1500 may optionally include the ODC circuit 200 or300 shown in FIGS. 2 and 3.

The interface device 1600 may be configured to transmit data to acommunication network or may receive data from the communicationnetwork. The interface device 1600 may include, for example, an antenna,a wired/wireless transceiver, and so on.

FIG. 7 is a block diagram of an embodiment of a user system including adepth information based optical distortion correction (ODC) circuit,according to aspects of the present invention.

Referring to the embodiment of FIG. 7, the user system 2000 may includea lens 2100, an image sensor 2200, an image signal processor (ISP) 2300,a display 2400, a lens positioning device 2500, a flash 2600, acontroller 2700, and a memory 2800.

The image sensor 2200 may photograph (or capture) an image, wherein theimage sensor 2200 may receive an optical image signal through a lens2100 and may convert the received optical image signal into anelectrical image signal. For example, the image sensor 2200 may includea charge coupled diode (CCD) image sensor or a complementary metal oxidesemiconductor (CMOS) image sensor.

The ISP 2300 may process the image signal photographed by the imagesensor 2200. The ISP 2300, including each of the ODC circuits 100 to 300according to some embodiments of the present invention, may correctoptical distortion in the image photographed by the image sensor 2200.The ISP 2300 including the ODC circuit 100 shown in FIG. 1 isexemplified in the embodiment of FIG. 7. However, the ISP 2300 mayoptionally include the ODC circuit 200 or 300 shown in FIGS. 2 and 3.

The display unit 2400 may display the image processed by the ISP 2300.The display 2400 may include, as examples, a liquid crystal display(LCD) or an organic light emitting diode (OLED) display panel. Theseshould be considered non-limiting examples.

The lens positioning 2500 device may be configured to adjust a focaldistance using an AF algorithm and may adjust the position of the lensaccording to the adjusted focal distance. The flash 2600, including alight source, a reflector, etc., may emit light. The controller 2700 maybe configured to control the overall operation of the user system 2000.

The memory 2800 may store data processed by the ISP 2300 and/or programsexecuted by the ISP 2300. The memory 2800 may include, for example, oneor more volatile memories, such as a double data rate static DRAM (DDRSDRAM), a single data rate SDRAM (SDR SDRAM) or a SRAM, and/or one ormore non-volatile memories, such as an electrical erasable programmableROM (EEPROM) or a flash memory. In various embodiments, the memory 2800may be combined with the user system 2000 in the form of a memory card.

In the embodiment of FIG. 7, the user system 2000 may further include aninterface 2900 configured to transmit data to a communication network orreceive data from the communication network. The interface 2900 mayinclude, for example, an antenna or a wired/wireless transceiver.

According to embodiments, the user system 2000 may be an arbitrarypick-up or handheld device, such as a digital camera, a compact camera,a high-end camera, or a digital single lens reflex (DLSR) camera.

FIG. 8 is a block diagram of an embodiment of another user systemincluding a depth information based optical distortion correction (ODC)circuit according to some embodiments of the present invention.

Referring to the embodiment of FIG. 8, the user system 3000 may includea central processing unit (CPU) 3100, a keyboard 3200, a display 3300, amemory 3400, a storage 3500, an image sensor 3600, an image signalprocessor (ISP) 3700, and a data bus 3800.

The CPU 3100, the keyboard 3200, the display 3300, the memory 3400, thestorage 3500, the image sensor 3600 and the ISP 3700 may be connected toeach other through the data bus 3800. The data bus 3800 may correspondto a path through which data moves, including various types of knownforms of data paths.

The CPU 3100, including a controller, an operation device, etc., mayexecute a program and may process data. In various embodiments, the CPU3100 may further include a cache memory positioned inside or outsideuser system 3000.

The keyboard 3200, including a plurality of keys, may receive letters ordigits from a user or may receive data through various function keys.The display 3300, including one or more displays, may display images. Insome embodiments, the keyboard 3200 and display 3300 may be combined inthe form of a touchscreen or the like, as an example.

The memory 3400 may include one or more volatile memories, such as adouble data rate static DRAM (DDR SDRAM), a single data rate SDRAM (SDRSDRAM) or a SRAM, as examples. The volatile memories may function as aworking memory for storing the data processed by the CPU 3100.Alternatively, the volatile memories may store the data processed by theISP 3700.

The memory 3400 may include one or more non-volatile memories, such asan EEPROM or a flash memory, as examples. The non-volatile memories maystore the programs executed by the CPU 3100 and/or the ISP 3700.

The storage 3500, including a recording media, such as a floppy disk, ahard disk, a CD-ROM, a DVD, or other non-transitory media, may storedata and/or programs.

The image sensor 3600 may photograph or capture an image. The imagesensor 3600 may receive an optical image signal through at least onelens and may convert the received optical image signal into anelectrical image signal. As examples, the image sensor 3600 may includea charge coupled diode (CCD) image sensor or a complementary metal oxidesemiconductor (CMOS) image sensor.

The ISP 3700 may process the image signal photographed by the imagesensor 3600. The ISP 3700, including one of the ODC circuits 100 to 300,as examples, may correct optical distortion in the image photographed bythe image sensor 3600. In this embodiment, the ISP 3700 includes the ODCcircuit 100 shown in FIG. 1, as exemplified in FIG. 8. However, in otherembodiments, the ISP 3700 may optionally include one of the ODC circuits200 or 300 shown in FIGS. 2 and 3.

Although not particularly shown in FIG. 8, the user system 3000 mayfurther include one or more other types of input devices, such as amouse, a button, a keypad, a touch screen or a microphone, and/or one ormore output devices, such as a speaker, projector, and so on.

In addition, the user system 3000 may further include an interface 3900transmitting data to a communication network or receiving data from thecommunication network. The interface may include, for example, anantenna or a wired/wireless transceiver.

According to various embodiments, the user system 3000 may be anarbitrary computing system, such as a mobile phone, a smart phone, apersonal digital assistant (PDA), a desktop, a notebook computer, or atablet PC, as examples.

FIG. 9 is a block diagram of still another user system including depthinformation based optical distortion correction (ODC) circuits accordingto some embodiments of the present invention.

Referring to the embodiment of FIG. 9, the user system 4000 may includea tuner 4100, a processor 4200, a display 4300, an image sensor 4400, animage signal processor (ISP) 4500, and a memory 4600.

The tuner 4100 may be configured to receive a broadcasting signal. Thetuner 4100 may receive an analog broadcasting signal or a digitalbroadcasting signal. Alternatively, the tuner 4100 may also beconfigured to receive a terrestrial broadcasting signal, a cablebroadcasting signal, or a satellite broadcasting signal, or combinationsthereof.

The processor 4200 may be configured to control the overall operation ofthe user system 4000.

The display 4300 may be configured to display the broadcasted signalreceived from the tuner 4100 or the image processed by the ISP 2300. Thedisplay 4300 may include, for example, a liquid crystal display (LCD) oran organic light emitting diode (OLED) display panel, or other displays.

The image sensor 4400 may be configured to photograph or capture animage. The image sensor 4400 may receive an optical image signal througha lens (not shown) and may convert the received optical image signalinto an electrical image signal. For example, the image sensor 4400 mayinclude a charge coupled diode (CCD) image sensor or a complementarymetal oxide semiconductor (CMOS) image sensor.

The ISP 4500 may process the image signal photographed by the imagesensor 4400. The ISP 4500, including one of the ODC circuits 100 to 300according to some embodiments of the present invention, may correctoptical distortion in the image photographed by the image sensor 4400.The ISP 4500 including the ODC circuit 100 shown in FIG. 1 as an examplein FIG. 9. However, in other embodiments, the ISP 4500 may optionallyinclude one of the ODC circuits 200 or 300 shown in FIGS. 2 and 3.

The memory 4600 may store data processed by the processor 4200 and/orthe ISP 4500 and/or programs executed by the ISP 4500. The memory 4600may also store the image photographed by the image sensor 4400. Thememory 4600 may include, for example, one or more volatile memories,such as a double data rate static DRAM (DDR SDRAM), a single data rateSDRAM (SDR SDRAM) or a SRAM, and/or one or more non-volatile memories,such as an electrical erasable programmable ROM (EEPROM) or a flashmemory, as examples.

According to embodiments, the user system 4000 may be an arbitrarybroadcast receiver, such as a cable receiver, digital television (TV),or a smart TV, as examples.

Hereinafter, an embodiment of a depth information based opticaldistortion correction (ODC) method according to aspects of the presentinvention will be described with reference to FIGS. 10 to 12. For thesake of convenience, detailed descriptions of already discussed subjectmatter will be omitted.

FIG. 10 is a flowchart illustrating an embodiment of a depth informationbased optical distortion correction (ODC) method according to aspects ofthe present invention, FIG. 11 is a flowchart specifically illustratingan embodiment of pre-processing shown in FIG. 10, and FIG. 12 is aflowchart specifically illustrating an embodiment of optical distortioncorrection shown in FIG. 10.

Referring to the embodiment of FIG. 10, in the depth information basedoptical distortion correction (ODC) method, pre-processing is firstperformed to generate a projection matrix (S410). The pre-processingwill be described in more detail with reference to FIG. 11.

Referring to the embodiment of FIG. 11, in the pre-processing (S410)step, at least two images having different depths are first acquired(S411). Here, the at least two images may be images of original gridspositioned at different distances. Different extents of opticaldistortion may be generated in the at least two images having differentdepths.

Next, the optical distortion in each of the at least two images iscalculated (S412). The at least two images may be compared with theoriginal grid image to calculate the optical distortion in each of theat least two images.

Next, depths of the at least two images are acquired (S413). The depthsof the at least two images may be acquired using a stereo depthextraction algorithm, an auto focus (AF) algorithm or a time of flight(TOF) sensor, as examples.

Next, a projection matrix is calculated using the calculated opticaldistortion in each of the at least two images and the acquired depths ofthe at least two images (S414), and the calculated projection matrix isstored (S415).

In some embodiments, as shown in FIG. 11, after acquiring the depths ofthe at least two images (S411), the optical distortion in each of the atleast two images may be calculated (S412). Optionally, in someembodiments, the calculating of the optical distortion (S412) and theacquiring of the depths of the at least two images (S413) may beperformed at the same time.

Next, referring again to FIG. 10, a distorted image is photographed (orcaptured) and received by an image sensor (S420), e.g., via a lens. Asexamples, the image sensor may include a charge coupled diode (CCD)image sensor or a complementary metal oxide semiconductor (CMOS) imagesensor. Next, optical distortion in the distorted image is corrected(S430). The correcting of the optical distortion will be described inmore detail with reference to FIG. 12.

Referring to the embodiment of FIG. 12, in the correcting of the opticaldistortion (S430), the depth of the distorted image is acquired (S431).The depth of the distorted image may be acquired using a stereo depthextraction algorithm, an auto focus (AF) algorithm or a time of flight(TOF) sensor, as examples.

Next, a correction grid corresponding to the depth of the distortedimage is dynamically generated (S432). Here, coordinates of thecorrection grid may be generated using coordinates of the original grid,depth information of the distorted image and the projection matrix.Coordinates of a grid may include coordinates of grid points (e.g.,x,y,z).

Next, the optical distortion in the distorted image is corrected usingthe correction grid (S433). The corrected image may be stored,transmitted, or otherwise output.

Those skilled in the art will appreciate that many variations andmodifications can be made to the preferred embodiments withoutsubstantially departing from the principles of the present invention.Therefore, the disclosed preferred embodiments of the invention are usedin a generic and descriptive sense only and not for purposes oflimitation. It is intended by the following claims to claim that whichis literally described and all equivalents thereto, including allmodifications and variations that fall within the scope of each claim.

What is claimed is:
 1. A depth information based optical distortioncorrection (ODC) circuit, comprising: a processor, the processorcomprising: a depth acquisition unit configured to acquire a depth of animage; a projection matrix calculation unit configured to calculate theprojection matrix using an optical distortion in at least two imagescaptured by an image sensor and having different depth information of atleast two images; a grid generation unit configured to dynamicallygenerate a correction grid corresponding to the depth of the image usingdepth information of the image and a projection matrix; and a distortioncorrection unit configured to correct optical distortion in the imageusing the correction grid.
 2. The ODC circuit of claim 1, wherein thegrid generation unit is configured to generate coordinates of thecorrection grid using coordinates of an original grid, the depthinformation of the image, and the projection matrix.
 3. The ODC circuitof claim 1, the processor further comprising: a distortion calculationunit configured to compare the at least two images with the originalgrid and to calculate the optical distortion of each of the at least twoimages.
 4. The ODC circuit of claim 1, wherein the depth acquisitionunit is configured to acquire the depths of the at least two images. 5.The ODC circuit of claim 1, further comprising: a storing unit storingthe projection matrix.
 6. The ODC circuit of claim 1, wherein the depthacquisition unit is configured to acquire a depth of at least one frameamong a plurality of frames captured by an image sensor.
 7. The ODCcircuit of claim 1, wherein the depth acquisition unit is configured toacquire the depth of the image using a stereo depth extractionalgorithm.
 8. The ODC circuit of claim 1, wherein the depth acquisitionunit is configured to acquire the depth of the image using an auto focus(AF) algorithm.
 9. The ODC circuit of claim 1, wherein the depthacquisition unit is configured to acquire the depth of the imageprovided from a time of flight (TOF) sensor.
 10. The ODC circuit ofclaim 1, wherein the depth acquisition unit is configured to acquire thedepth of the image input by a user.
 11. A depth information basedoptical distortion correction (ODC) method, the method comprising:acquiring at least two images having different depths using an imagesensor; calculating a projection matrix using optical distortion in theat least two images and depths of the two images; dynamically generatinga correction grid corresponding to the depth of the image using thedepth information of the image and a projection matrix; and correctingoptical distortion in the image using the correction grid.
 12. The ODCmethod of claim 11, wherein the dynamically generating of the correctiongrid corresponding to the depth of the image comprises dynamicallygenerating coordinates of the correction grid using coordinates of anoriginal grid, the depth information of the image, and the projectionmatrix.
 13. The ODC method of claim 11, wherein the pre-processingfurther comprises storing the projection matrix in a storage media. 14.An imaging system with depth information based optical distortioncorrection (ODC), comprising: a memory having stored therein aprojection matrix; an image sensor configured to receive an opticalimage signal from an image captured through at least one lens and toconvert the received optical image signal into an electrical imagesignal; an image signal processor configured to process the electricalimage signal into a distortion corrected optical image, the image signalprocessor comprising: a depth acquisition unit configured to acquire adepth of the image; a projection matrix calculation unit configured tocalculate the projection matrix using an optical distortion in at leasttwo images captured by an image sensor and having different depths, andthe depth information of the at least two images; a grid generation unitconfigured to dynamically generate a correction grid corresponding tothe depth of the image using depth information of the image and theprojection matrix; and a distortion correction unit configured tocorrect optical distortion in the image using the correction grid; and adisplay controller configured to output the corrected optical image to adisplay.
 15. The system of claim 14, wherein the grid generation unit isconfigured to generate coordinates of the correction grid usingcoordinates of an original grid, the depth information of the image, andthe projection matrix.
 16. The system of claim 14, further comprising: adistortion calculation unit configured to compare the at least twoimages with the original grid and to calculate the optical distortion ofeach of the at least two images.
 17. The system of claim 14, wherein thedepth acquisition unit is configured to acquire the depth of the imageusing an auto focus (AF) algorithm; a time of flight (TOF) sensor; or byan input by a user.